Repository navigation
feat(core): expose aiohttp pool wait telemetry - #3632
Merged
Merged
Conversation
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-3632-a8c03d75dd1f.docs.buildwithfern.com/nemo/gym |
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Contributor
Author
|
/ok to test 9b3f3d2 |
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Contributor
Author
|
/ok to test 6c46eae |
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Contributor
Author
|
/ok to test db5d90e |
athitten
marked this pull request as ready for review
September 23, 2026 22:12
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Oct 1, 2026
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Contributor
Author
|
/ok to test d6b15ff |
ananthsub
previously approved these changes
Oct 1, 2026
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Contributor
Author
|
/ok to test a8c03d7 |
ananthsub
approved these changes
Oct 1, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36940078210 |
7 tasks done
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Oct 13, 2026
## What does this PR do? Follow-up to NVIDIA-NeMo#3632, split out as requested in NVIDIA-NeMo#3632 (comment). In `_request_with_retries()`, the generic `except Exception` branch handles every error except `ServerDisconnectedError` and `ClientOSError`, which have their own branches. This includes aiohttp timeouts. When the caller does not pass `_max_num_tries`, this branch counted tries only for external callers. So an internal caller (`_internal=True`) that set `_max_connection_retries` to 2 or more never reached the cap and retried forever. NVIDIA-NeMo#3415 already counts tries for callers that pass `_max_num_tries`. This PR fixes the remaining path, where `_max_num_tries` is not passed. - Count tries for internal callers too, so `_max_connection_retries` stops them. - Keep everything else the same: - internal callers with no cap still retry with no limit and print no retry message, as NVIDIA-NeMo#90 intended; - external callers still stop at `MAX_NUM_TRIES`; - callers that pass `_max_num_tries` are unchanged. - Update the comment that said internal calls are not counted. It keeps NVIDIA-NeMo#90's reason for not limiting internal calls. - Add three tests: - the cap stops internal callers, with tracing off and on; - internal callers with no cap retry silently past `MAX_NUM_TRIES`; - external callers stop at `MAX_NUM_TRIES` and print one retry message per try. - The cap test, `test_request_retry_cap_bounds_internal_callers`, replaces `test_retry_cap_bounds_internal_callers`, which was removed from NVIDIA-NeMo#3632. - Make the retry tests fail instead of hang when the loop never stops: - The test helper `_mock_global_client` (added in NVIDIA-NeMo#2383) now takes the error to raise. - Its fake `request()` calls `pytest.fail()` once it is called more often than any test needs. `pytest.fail()` raises a `BaseException`, which the loop's `except Exception` does not catch. So the test fails at once, even if the loop never sleeps. - Two NVIDIA-NeMo#3415 tests now use this helper instead of their own mocks: `test_explicit_transport_attempt_cap_disables_request_retry` and `test_request_caps_generic_errors_at_the_lower_attempt_limit`. Before, if the generic branch ignored `_max_num_tries`, the first test hung, sleeping a real 0.5 s per retry. No caller in this repo sets both `_internal=True` and `_max_connection_retries` today: - The internal callers pass no cap: `ServerClient`, the simple agent's direct tool calls to Resources servers (`tool_access`), the Terminus 2 sandboxed agent's model client (`NeMoGymAsyncOpenAI(internal=True)`), the OpenHands capture route, and the LiteLLM version check. - The callers with a cap are not internal: the remote agent and sandbox transport use 1, Tavily search uses 0, and vLLM with `endpoint_file` uses a default of 8. - The OpenAI model server and the Gymnasium agent pass `_max_num_tries` only when `upstream_max_num_tries` or `model_server_transport_max_attempts` is set. Neither passes `_max_connection_retries`, so neither changes. Related: - NVIDIA-NeMo#3553 added the cap check to this branch without counting internal tries. - NVIDIA-NeMo#3415 added `_max_num_tries` and counts tries on that path. - NVIDIA-NeMo#3159 tracks bounding every retry by default, including internal callers with no cap. This PR only makes an explicit cap work for them. - NVIDIA-NeMo#2216 tracks the `ServerDisconnectedError` and `ClientOSError` branches, which this PR does not change. - Open PR NVIDIA-NeMo#2935 adds a timeout branch that copies the old counting, so internal callers there still do not count tries. Whichever PR merges later must count tries for every caller. ## Validation All runs used an activated venv from `uv sync --extra dev --extra sandbox --extra openshell --extra telemetry --extra mlflow --extra wandb` (the extras in `full-test-suite.yml`). Every installed package matches `uv.lock` (192 packages, including aiohttp 3.14.4, nemo-lens 0.2.0+b0f977d, and pytest 9.0.3). - `pytest tests/unit_tests/test_server_utils.py responses_api_agents/remote_agent/tests/test_app.py tests/unit_tests/telemetry/`: 465 passed. - With the old code restored, `test_request_retry_cap_bounds_internal_callers` fails in under a second with `Failed: retry loop did not stop` instead of hanging. It fails the same way when the retry sleep is also removed. - Each change below makes at least one test fail. Each run of `test_server_utils.py` took about 3 s, and none hung: - the cap is ignored; - internal callers are limited by `MAX_NUM_TRIES`; - internal callers print; - external tries are double-counted; - external callers stop printing the retry message; - the `MAX_NUM_TRIES` raise is removed; - the retry sleep after a generic error is skipped; - the cap check ignores `_max_num_tries` attempts; - the generic branch ignores `_max_num_tries`; - the traced path drops `_internal` or `_max_connection_retries`. - `pytest -n 16 tests/unit_tests responses_api_agents/remote_agent/tests`: 8089 passed, 5 skipped. The per-test results match `main` at a3cf403 (8085 passed, 5 skipped). The 4 extra results are the new tests, with the cap test run twice (tracing off and on). - The same focused tests pass on a local merge with the latest `main` (NVIDIA-NeMo#4300), which changes no package versions. - `pre-commit run --all-files`: passed. ## Rollouts N/A. This changes a retry counter in the shared HTTP helper. No caller in this repo changes behavior, and no model, agent, environment, or verifier logic is involved. ## Compatibility and benchmark impact None for callers in this repo. A caller that sets `_internal=True` and `_max_connection_retries` to 2 or more, without `_max_num_tries`, now stops after that many tries on errors that reach the generic `except Exception` branch, including timeouts, instead of retrying forever. This includes code outside this repo that uses `NeMoGymAsyncOpenAI(internal=True, max_connection_retries=N)`. `ServerDisconnectedError` and `ClientOSError` keep their own count, as before. ## Checklist - [x] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. N/A: internal helper behavior with no user-facing docs. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). - [x] New source files include the required Apache-2.0 SPDX header. N/A: no new files. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Abhishree <athittenaman@nvidia.com>
8 tasks
7 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Closes #3008.
queue_duration_msand all-attemptconnect_totalinstruments undergym.http.connection_pool, labelled withnemo.gym.http.destination.server.name.QueueTimedTCPConnectoronly during metrics export; otherwise plainTCPConnector. No poolTraceConfigoverhead.ok/abandoned.0 = unlimited, rejects positive limits that divide to zero, and routes remote-agent POSTs through the shared wrapper without replaying them.Validation
Acceptance criteria:
connect_totaldistinguishes queued/immediate acquisitions; constraints:total,per_host,unknown.Real-connector regressions cover timeout/retry, cancellation, queued redirects, first-request/
ServerClientlabels, CLIENT spans, and aiohttp method/signature compatibility. Independent reviews checked code/tests and benchmark methodology/results.Checks on
e74b8e865:.venv/bin/pytest tests/unit_tests/test_server_utils.py tests/unit_tests/telemetry tests/unit_tests/test_opensandbox_http_transport.py tests/unit_tests/test_fern_docs_links.py -q -o addopts='' -o log_cli=false— 398 passed, 1 skipped, 17 subtests passed..venv/bin/pre-commit run --all-files— passed.git diff --check— passed.cd fern && npm run check -- --warnings— 0 errors; authenticated redirect checking was skipped without a Fern token.Rollouts
N/A for model compute: shared HTTP observability/transport changes, not model or verifier decisions. Real aiohttp services exercise the affected request paths.
Compatibility and benchmark impact
No config/dataset migration;
0stays unlimited. Budgets apply per server/CLI process group, not deployment-wide. Export must precede client creation; existing clients are not retrofitted. Dashboard/log queries should use the destination attribute andper_worker_per_hostfield. Histogram percentiles are queued-only.The issue benchmark plan was rerun on
e74b8e865: 10 ms loopback backends, concurrency 128/256/512, 1/4/16 client-worker processes, all three pool-sizing cases. Four interleaved settings (off, CLIENT spans only, metrics only, both), five repetitions: 540 cells, 1,873,920 measured requests, zero errors. Recorded queue/client-span/backend p50/p95/p99, throughput, sockets, FDs, errors. Warmup excluded; counter deltas matched requests. Total/adequate cases used four host/port destinations; per-host used one.Four workers, concurrency 128, metrics+CLIENT spans enabled. Worker total/per-host limits: adequate 64/64, per-host 32/8, total 8/32. Latencies: ms, medians of five round-specific percentiles.
Sockets/FDs sum worker peaks sampled every 5 ms, not simultaneous deployment peaks. Raw queue capture supplements SDK recording here; CPU is measured separately.
Throughput ranges: 6,927–10,908 / 2,546–2,802 / 3,273–3,717 requests/s respectively. One adequate round had CLIENT-span p99 71.87 ms. Total-limit queueing ranged 4.77–5.86%, making all-request p95 vary 12.65–125.91 ms. aiohttp's total limit is a soft scheduling limit, not a strict acquired-connection/socket cap across destinations; this case measures intermittent waits, not steady queueing behind a hard cap.
Isolated latest-branch CPU comparison: six interleaved rounds, concurrency 32, separate pinned client/backend cores, real SDK/in-memory reader; no CLIENT spans, queue observer, or FD sampler. Immediate requests used 8,192 measured + 1,024 warmup; undersized cases 2,048 + 256 with 10 ms backends. 54 cells, 221,184 measured requests, zero errors. CPU ranges are round minima/maxima; p95 covers full response reads.
Matching-main baseline
12399ad81: immediate CPU 58.21 (57.67–58.41) µs/request. PR metrics-off median is +0.30% with overlapping ranges, not proof of zero cost. Enabled CPU cost: +2.96% immediate; +15.28% per-host (71.17% queued); +1.22% total, where only 0.78% queued (0.63–0.98%). The total case is not sustained-queue cost. No universal throughput/latency improvement is claimed.Environment: AMD Ryzen Threadripper 3960X, Linux, Python 3.13.14, aiohttp 3.14.1, OpenTelemetry SDK 1.44.0. Network exporter/periodic collection, DNS/TLS/proxies, model throughput, and memory/allocation costs were not measured. Benchmark scripts and raw evidence remain local and are not committed.
Checklist